Conrail Performance Guide v3.2

Ethernet minimum frame size is 64 Bytes. When Ethernet frames are sent onto the wire, Inter Frame Gap and Preamble bits are added. Minimum size of Ethernet frames on the physical layer is 84 Bytes (672 bits).

For a 10 Gbit/s interface, the number of frames per seconds can reach up to 14.88 Mpps for traffic using the smallest Ethernet frame size. It means a new frame will have to be forwarded each 67 ns.

A CPU running at 2Ghz has a 0.5 ns cycle. Such a CPU has a budget of only 134 cycles per packet to be able to process a flow of 10 Gb/s.

Generic Linux Ethernet drivers are not performant enough to be able to process such a 10Gb/s packet flow.

Indeed lots of times are required to :

A SDN solution like Contrail needs to use specific setup and mechanisms in order to be able to process network flows onto a generic x86 platform at a high rate.

Virtio was developed as a standardized open interface for virtual machines (VMs) to access simplified devices such as block devices and network adaptors.

There are two parts to networking within VirtIO:

In Contrail Networking, VirtIO is used to connect with guest VM vNIC onto the vrouter vif interface. In the diagram below you have a detailed view of a Virtual instance connectivity with a Kernel mode vrouter :

For more information on VirtIO :

A traditional server has a single CPU, a single RAM and a single RAM controller.

A RAM can be made of several DIMM banks in several sockets, all being associated to the CPU. When the CPU needs access to data in RAM, it requests it to its RAM controller.

Recent servers can have multiple CPUs, each one having its own RAM and its own RAM controller. Such systems are called NUMA systems, or Non-Uniform Memory Access. For example, in a server with 2 CPUs, each one can be a separate NUMA: NUMA0 and NUMA1.

When CPU0 needs to access data located in RAM0, it will go through its local RAM controller 0. Same thing happens for CPU1.

When CPU0 needs to access data located in the other RAM1, the first (local) controller 0 has to go through the second (or remote) RAM controller 1 which will access the (remote) data in RAM 1. Data will use an internal connection between the 2 CPUs called QPI, or Quick Path Interconnect, which is typically of a high enough capacity to avoid being a bottleneck, typically 1 or 2 times 25GBps (400 Gbps). For example the Intel Xeon E5 has 2 CPUs with 2 QPI links between them; Intel Xeon E7 has 4 CPUs, with a single QPI between pairs of CPUs.

The fastest RAM that the CPU has access to is the register, which is inside the CPU and reserved to it.

Beyond the register, the CPU has access to cached memory, which is a special memory based on higher performance hardware.

Cached memory are shared between the cores of a single CPU. Typical characteristics of memory cache are:

If the CPU needs to access data that is in the main RAM, it has to use its RAM controller.

Access to RAM takes then typically 170 CPU cycles (the green line in the diagram). Access to the remote RAM through the remote RAM controller typically adds 200 cycles (the red line in the diagram), meaning RAM latency is roughly doubled.

When data needed by the CPU is located both in the local and in the remote RAM with no particular structure, latency to access data can be unpredictable and unstable.

A single physical CPU core with hyper-threading appears as two logical CPUs to an operating system.

While the operating system sees two CPUs for each core, the actual CPU hardware only has a single set of execution resources for each core.

Hyper-threading allows the two logical CPU cores to share physical execution resources.

The sharing of resources allows two logical processors to work with each other more efficiently, and allows a logical processor to borrow resources from a stalled logical core (assuming both logical cores are associated with the same physical core). Hyper-threading can help speed processing up, but it’s nowhere near as good as having actual additional cores.

The performance of vRouter with sibling HT cores can increase by 10% to 20% (result is based on performance tests described hereinafter).

Memory is managed in blocks known as pages. On most systems, a page is 4Ki. 1Mi of memory is equal to 256 pages; 1Gi of memory is 256,000 pages, etc. CPUs have a built-in memory management unit that manages a list of these pages in hardware.

The Translation Lookaside Buffer (TLB) is a small hardware cache of virtual-to-physical page mappings. If the virtual address passed in a hardware instruction can be found in the TLB, the mapping can be determined quickly. If not, a TLB miss occurs, and the system falls back to slower, software based address translation. This results in performance issues. Since the size of the TLB is fixed, the only way to reduce the chance of a TLB miss is to increase the page size.

Virtual memory address lookup slows down when the number of entries increases.

A huge page is a memory page that is larger than 4Ki. In x86_64 architecture, in addition to standard 4KB memory page size, two larger page sizes are available: 2MB and 1GB.

Contrail DPDK vrouter can use both or only one huge page size.

Data Plane Development Kit (DPDK) is a set of data plane libraries and network interface controller drivers for fast packet processing, currently managed as an open-source project under the Linux Foundation.

The main goal of the DPDK is to provide a simple, complete framework for fast packet processing in data plane applications.

The framework creates a set of libraries for specific environments through the creation of an Environment Abstraction Layer (EAL), which may be specific to a mode of the Intel® architecture (32-bit or 64-bit), Linux* user space compilers or a specific platform.

These environments are created through the use of make files and configuration files. Once the EAL library is created, the user may link with the library to create their own applications.

The DPDK implements a "run to completion model" for packet processing, where all resources must be allocated prior to calling Data Plane applications, running as execution units on logical processing cores.

The model does not support a scheduler and all devices are accessed by polling. The primary reason for not using interrupts is the performance overhead imposed by interrupt processing.

For more information please refer to dpdk.org documents http://dpdk.org/doc/guides/prog_guide/index.html

With DPDK there is a direct link between application data stored in host memory and the NIC memory used to transfer data onto the wire :

DPDK uses message buffers known as mbufs to store packet data into the host memory. These mbufs are stored in memory pools known as mempools. Mempools are set up as a ring, which creates a pool with a configuration similar to a first-in, first-out (FIFO) system.

Rings descriptors are managing data storage into mempools. The more descriptors RX/TX rings are containing, the more memory size will be required in each mempool to store data.

The Host OS exchanges packets with the NIC through the so called rings. A ring is a circular array of descriptors allocated by the OS in the system memory (RAM). Each descriptor contains information about a packet that has been received or that is going to be transmitted.

RX ring

RX ring is managing Data transfer from NIC memory to host memory :

Synchronization between the OS and the NIC happens through two registers, whose content is interpreted as an index in the RX ring:

TX ring

TX ring is managing data transfer from host memory to NIC memory :

Synchronization between the host OS and the NIC happens through two registers, whose content is interpreted as an index in the TX ring:

DMA

Direct Memory Access (DMA) allows PCI devices to read (write) data from (to) memory without CPU intervention. This is a fundamental requirement for high performance devices.

When DPDK is used, Network interfaces are no more managed in Kernel space. Legacy NIC driver which is usually used to manage the NIC has to be replaced by a new driver which is able to run into user space. This new drive, called Poll Mode Driver (PMD) will be used to manage the network interface into user space with the DPDK library.

A Poll Mode Driver consists of APIs, provided through the BSD driver running in user space, to configure the devices and their respective queues. In addition, a PMD accesses the RX and TX descriptors directly without any interrupts (with the exception of Link Status Change interrupts) to quickly receive, process and deliver packets in the user’s application.

Some PMD are being used to manage physical interfaces :

Some PMD are being used to manage virtual interfaces :

In Linux user space environment, the DPDK application runs as a user-space application using the pthread library.

PCI information about devices and address space is discovered through the /sys kernel interface and through kernel modules such as uio_pci_generic, igb_uio or vfio-pci.

Different PMDs may require different kernel drivers in order to work properly. Depending on the PMD being used, a corresponding kernel driver should be loaded and bound to the network ports. Before loading, make sure that each NIC has been flashed with the latest version of NVM/firmware.

Use lstopo and numactl commands to check cores numbering and numa nodes architecture

To check NIC assignment to NUMA node. We recommend assigning the Cores (real and sibling/hyperthreaded) to the same NUMA as the NIC(s).

Use lspci command to find PCI addresses of NICs that will be bound to DPDK process

Use dpdk_nic_bind.py command to verify proper binding of NICs to DPDK

vRouter is made of 2 parts :

vRouter agent is used to manage the communication between Contrail SDN Controller and vRouter.

vRouter agent has 2 interfaces :

vRouter agent is always running in compute node user mode.

vRouter dataplane has 2 kinds of interfaces

vRouter dataplane is running either in Linux Kernel space or in Linux User space when DPDK is used. Better performances are expected when vRouter dataplane is running into Linux User space.

In order to get good network performances, virtual instances (especially VNF - Virtual Network Function) are implementing DPDK. 4 topologies are possible:

An all DPDK stack, used both at vrouter level and at VNF level, is providing the best performance.

When DPDK is used into the virtual instance, vNIC driver (virtio-net) used for packet processing in Kernel space is replaced by a virtio Poll Mode Driver (virtio-net PMD) in order to perform packet processing in Virtual Instance user space.

There are 4 main steps in packet processing :

DPDK vrouter is a multi-threads application.

There are 3 kinds of threads into a DPDK vrouter :

The term lcore refers to an EAL (Environment Abstraction Layer) thread, which is really a Linux/FreeBSD pthread (physical Thread). A numbering into the dpdk vrouter is used for lcores.

DPDK control threads can be pinned on any CPU. These CPUs are only used during Contrail vRouter startup for DPDK initialization. As they did not require high CPU capacity, a good strategy is to use the CPU allocated to the Operating System for these threads.

This lcore numbering used in vrouter can be seen into source file (vr_dpdk.h).

An enumeration is defining this numbering :

0 to 9 lcore numbers are statically defined into the source code and can’t be configured by user.

lcore numbers 10 and above are used for forwarding purpose and defined with CPU affinity value. This value is set into contrail-vrouter-dpdk.ini configuration file.

For instance 0x000154000154 CPU affinity is giving following mapping :

it means Host CPU numbers 2, 4, 6, 8, 26, 28, 30 and 32 are used for vrouter forwarding logical cores.

When the router boots up, it displays a message about its logical cores numbering. For instance, we can get into contrail-vrouter-dpdk-stdout.log file, such a message :

This is this vrouter logical core numbering that has to be in dropstats command (not the real host CPU numbering)

A maximum of 16 polling cores is currently supported by Contrail as the maximum RX queue is currently 16:

DPDK vrouter forwarding plane process is made up of several kinds of threads:

Each set of threads is made up of several single threads:

Service and Processing threads are named lcore-slave-<ID>. Contrail vRouter cores ID have a specific meaning defined in the following C enum data structure:

We can find those names by running using the "ps" command with some additional arguments (this is from a compute node where vrouter is using 8 logical forwarding cores, 4 phy cores using HT siblings):

Here we have :

To list all lightweight processes created by contrail-vrouter-dpdk run following command:

The following command can be used. It provides in the last column the CPU load generated by each thread:

Using pidstat command we can see that some vrouter threads are running at 100% CPU. Those are the forwarding threads, the ones pinned based on the configured coremask :

In the example above, we can notice:

In order to list CPU cores assigned to contrail-vrouter-dpdk, we have to run taskset command for each lightweight process.

Starting from Contrail 3.2.5 DPDK library 17.02 is used. Earlier Contrail versions were based on DPDK library version 2.1 https://www.juniper.net/documentation/en_US/contrail3.2/information-products/topic-collections/release-notes/jd0e36.html#jd0e185

Contrail DPDK library fork

https://github.com/Juniper/contrail-dpdk

DPDK release notes

http://dpdk.org/doc/guides-17.02/rel_notes/index.html

DPDK version used by the vrouter can be checked into dpdk vrouter log files :

When the traffic is coming from the underlay network, encapsulated packets are received DPDK compute Physical interface which is bound to vrouter vif 0/0:

Incoming packets are processed in 3 steps :

Two main situations have to be considered for processing core selection :

vRouter forwarding threads are also responsible for polling virtio interface queues. The decision which thread will poll which virtio queue is made by Contrail DPDK vRouter.

If VM uses multiqueue then more vRouter forwarding threads (CPU cores) will be engaged in receiving packets from a single VM.

To avoid interrupts raised by DPDK vRouter (required to raise packet processing by QEMU/KVM) that influence packets processing Juniper recommends to only connect DPDK VMs and DPDK interfaces to DPDK vRouter.

Putting non-DPDK VMs on a DPDK node is affecting performance of both VM and vRouter:

As a result of these, the performance numbers would be the similar as that of kernel vRouter.

These are the guidelines from Intel with regard to Intel NIC X710 (in order not to bump into a known issue that impacts data plane)

It is required to have Contrail qemu / libvirt / nova packages installed on DPDK nodes in order to:

The following kernel parameters configure huge pages:

After adding the configuration to a kernel and rebooting server, the number of allocated huge pages can be checked from command line. The most common way of doing it is to check /proc/meminfo. This is not a good way, because it shows only one (default) huge page size.

ping: 3.2 lab

ping: 1908 lab

Since it is possible to configure multiple sizes of hugepages, the only source of information is /sys filesystem (for each numa node and each size - total/free):

To finish configuration of hugepages, hugetlbfs pseudo filesystem needs to be mounted. The following line needs to be added to /etc/fstab:

As mentioned before, DPDK vRouter needs its own hugepages. The DPDK library auto-detects the huge pages from the hugetlbfs mount point.

2GB for vRouter are based on setup provided in configuration file.

In order to allocate memory on the first NUMA socket, we are using option with only one parameter:

In order to allocate memory on the NUMA0 and NUMA1 socket, we are using option with only two parameters:

On a 2 NUMA nodes we have to allocate memory on each of them even if vRouter is only pinned on a single one. Indeed Virtual machines are pinned on both and require memory on each.

It is important to allocate hugepage memory to all NUMA nodes that will have DPDK interfaces associated with them. If memory is not allocated on a NUMA node associated with a physical NIC or VM, they cannot be used.

We can get all PCI Ethernet devices installed on the host using the following command :

We can check the PCI device related numa node id using one of following commands:

Using numactl we can get CPU IDs on each NUMA socket:

ping: best tool for numa topology is hwloc-ls GUI.

If only 1G huge pages are configured on the compute node, pagesize parameter must be specified into huge page table mount point in order for the vrouter to be aware that only 1G huge pages are available on the compute node:

Cf: https://github.com/Juniper/contrail-vrouter/blob/R1908/dpdk/vr_dpdk_table_mem.c#L80-L84

If this parameter is not present in the mountpoint, Contrail vrouter is assuming that 2MB hugepages are available, and will requests 2M huge pages.

If only 1GB huge pages are available onto the compute node, vrouter will fail to start.

When huge pages mount point is configured without pagesize parameter, vrouter will request 2M hugepages allocation.

Here we have specified huge pages mount with pagesize=1G parameter. vrouter will request 1G hugepages allocation at startup.

Disabling IRQ Balance is not recommended as it has no impact on performance. In RHEL, tuned’s cpu-partitioning profile leverages IRQ CPU affinity (isolated_cores).

stop irqbalance daemon and remove irqbalance from system startup:

The /etc/sysconfig/irqbalance configuration file contains a parameter named IRQBALANCE_BANNED_CPUS that allows to define CPUs to be excluded from consideration by the IRQ balancing service.

The following mask excludes CPUs 8 to 15 as well as CPU 33 from IRQ balancing:

(*) these features are requiring mergeable buffers to be supported.

When planning physical CPU cores allocation, the following aspects need to be taken into account:

If HT (Hyper Threading) is enabled, the first half of each numa node core are physical cores, and the second half their HT sibling.

ping: see hwloc-ls GUI

The best way to utilize all cores in the system, especially when using hw:cpu_policy=dedicated (see CPU isolation mechanisms) to use both siblings when calculating mask for vrouter (and other resources line systemd). In that case, both siblings for each core used for VM pinning can be utilized.

With OpenStack Having a core without its sibling listed in vcpu_pin_set variable in nova.conf file and hw:cpu_policy=dedicated set in flavor properties used to spawn an instance leads to a scheduling error (variable sibling_set referenced before assignment).

In order to get a CPU booked for a subset of tasks, we have to inform the Operating System scheduler not to use these CPUs for all the tasks it has to run.

These CPUs are told: "isolated" because they are no more used to process all tasks. In order to get a CPU isolated several mechanisms can be used:

RedHat Linux is currently supporting 2 different CPU isolation methods. They are isolcpus and tuned CPU partitioning, They can be used either independently or simultaneously.

A first method, isolcpus kernel parameter, has been proposed to keep CPUs away from linux scheduler. This isolation mechanism will:

The main drawbacks of using isolcpus are :

A second method, tuned CPU partitioning, has been proposed more recently. since almost all processes are started by systemd, physical CPU cores assignment can be configured in a different way:

This isolation mechanism will:

The main drawbacks of using tuned partitioning are :

In few words, tuned partitioning is more flexible and featureful than isolcpus, but is providing a lower CPU isolation. This is why both mechanisms are often used altogether to enforce CPU isolation.

In order to get DPDK vrouter threads pinned to a subset of CPUs on each compute node we have to define:

These values: CPU_LIST, DPDK_CTRL_THREAD_MASK and SERVICE_CORE_MASK are CPU masks used for CPU pinning with taskset command.

Currently only CPU assigned to packet processing threads (CPU_LIST / COREMASK) can be managed with provisioning tools.

Since version 20.03, we can also manage CPU assigned to control and service threads (DPDK_CTRL_THREAD_MASK and SERVICE_CORE_MASK) with provisioning tools.

Contrail can be installed through different tools: Contrail Cloud, Contrail Networking integrated with RHOSP and TripleO deployer or Juniper Contrail Ansible Deployer to name some.

The network performance does not scale as the number of vCPUs increases.

Guest cannot transmit or retrieve packets in parallel as virtio-net have only one TX and RX. virtio-net drivers must be synchronized before sending and receiving packets.

It is useless to assign more CPU to vrouter in order to increase packet processing rate, as only one vrouter CPU is used to poll the single vNIC queue.

Moreover, all vrouter CPU could have a packet to transmit to the vNIC single queue which is becoming a congestion point.

If we add more CPU, we increase the pressure on the vNIC single queue and it could lead to jitter and even packet drop (buffer overflow).

When multiqueue is used on a Guest VM, the vrouter will only be able to sustain guest vNIC configured with a number of queue below or equal to the number for processing core allocated to the vrouter.

In the ideal scenario, each multiqueue vNIC is configured with the same number of queue than the number of CPU allocated to vrouter for packet processing.

The difference with non-DPDK is that vRouter runs in the user space instead of host kernel.

Also there is a vhost userspace process wherein the vRouter copies packets to/from the VM.

Every queue inside the VM maps to a virtio ring on the host user space.

Some of the offload capabilities, e.g. GRO (Generic Receive Offload), is not supported with multiqueue virtio on DPDK vrouter in Contrail 3.2.

In this case vRouter runs as a multi-core process that exists on all assigned cores.

There is also a scale-out approach to packet processing using multiple cores so the performance of one VM is NOT limited by the performance of 1 core.

A vRouter might have more cores than there are queues in the VM. Queues can only be shared when the vRouter cores send to the VM queue.

When receiving from the queue, exactly one vRouter core will read from a queue (i.e. receive queues are not shared, in order to prevent packet reordering).

vNIC queues are not configured by Contrail vRouter, they are managed by OpenStack. Rqueue Size are defined at OpenStack level (using libvirt) : /etc/nova/nova.conf

Nova configuration file values are provisioned with TripleO.

vNIC queues size can be defined using NovaLibvirtRxQueueSize and NovaLibvirtRxQueueSize environment parameters. By default they are configured with a 512 value.

Nova configuration file is configured with <tripleo-templates-root-dir>/puppet/services/nova-compute.yaml file:

To enable multiqueue with DPDK in a Glance image, we add metadata hw_vif_multiqueue_enabled="true":

Number of queues to be created can’t be specified with Glance metadata. Currently, the number of queues will match the number of vCPUs, defined for the instance.

Nova is configuring automatically the number of queues of each vNIC. There is no way to manually define the number of queues to be created for each vNIC. We can only change the number of queues of a given vNIC after the virtual instance has been started, using some Linux commande into the virtual machine whose vNIC configuration has to be modified (cf next section).

Nova is using vif.py script to create instance virtual interfaces (https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py)

Number of configured queues for a vif is depending on:

When a Linux Kernel 3.x is used (RHOSP 13), a maximum of 8 queues are configured by Nova for each vif.

When a Linux Kernel 4.x is used (RHOSP 16), the same algorithm is used but using 256 instead of 8 as a maximum of queues to be configured by each vif.

Inside an guest instantiated guest with multiqueues enabled, the NIC channel setup can be checked and changed as needed with the commands below, inside each virtual machine.

In order to see the current number of queues we have to use :

In order to change the number of queues we have to use :

For instance, below we are configuring 4 logical queues on eth0 vNIC :

This should be done during interface initialization, for example in a "pre-up" action in /etc/network/interfaces. But it seems to be possible to change this configuration on a running interface without disruption.

Using virsh dumpxml command, we can verify queue size configured by Nova with libvirt :

Here, 5 queues have been allocated to the vNIC, rx/tx_queue_size is configured as 1024.

Juniper Networks recommends to tune performance in the following four areas:

The following performance optimization actions are details below:

Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report. Select Performance as the CPU Power and Performance policy.

Disable Turbo Boost to ensure the performance scaling increases with the number of cores.

Set memory frequency to the highest available number, NOT auto.

For optimal performance, bond mode must be set to 4, and hash set to layer 3/4 in contrail-vrouter-dpdk.ini on the physical interface (Since Contrail 5.0 release and later, Contrail is now containerized and DPDK parameters are no more defined into contrail-vrouter-dpdk.ini file but are installed by /bin/bash /entrypoint.sh into contrail-vrouter-agent-dpdk container - cf appendix section).

Then run service supervisor-vrouter restart for the changes to take effect.

Every packet sent by a compute node to a given remote compute node will use the same outgoing Ethernet interface for a given encapsulation protocol (VxLAN, MPLSoUDP or MPLSoGRE).

Using Bond linux setup "lacp bond mode=4,xmit_policy=l34” is performing load-balancing on layer3+layer4 outer header. With this setup when a UDP encapsulation protocol is used (MPLSoUDP or VxLAN), outgoing packets to be sent to a given compute node are spread onto both outgoing Ethernet interfaces according to UDP source port that is used to create traffic “entropy" (diversity) between different flows.

Received packets in Fortville NICs are processed differently to Niantic. Contrail vRouter adjusts to Intel recommendations and increases the number of TX and RX buffer descriptors from 128 to 2048 and the mbuf pool from 16K to 32K bytes.

Starting from the Contrail 3.2.10 release it will be possible to configure the number of TX / RX descriptors in contrail-vrouter-dpdk.ini file https://bugs.launchpad.net/juniperopenstack/+bug/1743391

Since Contrail 5.0 release and later, Contrail is now containerized and DPDK parameters are no more defined into contrail-vrouter-dpdk.ini file but are installed by /bin/bash /entrypoint.sh into contrail-vrouter-agent-dpdk container - cf appendix section.

Caveat: The Intel NIC may silently drop packets in case of a shortage of RX descriptors. The DPDK vRouter polls NIC for these statistics.

In non-multiqueue VNF with one DPDK interface, increasing the number of CPU cores for vRouter can actually decrease performance because of One to Many CPU cores queues are sending packets one by one what takes time and can produce microbursts.

On the diagram cores one by one are sending traffic to VM and when last finish data processing potentially first will have no time (buffers) to send VM (virtio queue).

During this case microbursts affecting VNF will be observed on vRouter as TX errors.

DPDK vrouter dimensioning is depending on following parameters:

When we are designing an OpenStack Contrail infrastructure we have to define how many CPU will be used for vRouter needs, how many for virtual instances and how many will remain to be used by the Operating system.

Service threads are used into the communication between vRouter dataplane (DPDK vrouter process) and the control plane (vrouter agent). They are involved in vNIC plugin to the vRouter, flow creation/deletion onto the vrouter. So we have two scenarios to consider :

One single CPU will be enough for Service Threads. If hyperthreading is enabled, a single logical core would be enough. This CPU could be chosen on any NUMA node, e.g. NUMA#1

We are allocating a single Physical CPU to services threads. If hyperthreading is enabled, both logical cores will be allocated to Service Threads. If lots of flows are required at a high rate, an additional CPU could be needed. These CPUs could be chosen on any NUMA node (but on the same NUMA).

We are running most of our vNIC in packet mode. Only a few of them will be using flow mode. We will allocate a Physical CPU on NUMA1 for vrouter Service Threads (as our forwarding cores have been allocated on NUMA0). We will use both logical cores on this physical CPU for our Service Threads.

Onto a Contrail OpenStack platform we have following kind of processes that will use CPUs:

We are defining 4 CPU groups:

We are applying these allocation rules to get the best performance as possible:

In the following example, we will describe a performance tuning configuration using a node with 36 physical cores split on 2 NUMA and Hyperthreading enabled (72 logical cores).

On this system we are applying the following typical CPU allocation:

Configuration described below is really relevant when a Contrail 20.03 and later release is used. CPUs 2-35,38-71 will be allocated to Contrail DPDK vRouter and to Nova. They will be isolated using both tuned partitioning and isolcpus mechanisms. The diagram below is describing the target situation:

In best case scenarios the vRouter can achieve 1,5Mpps per physical core.

The actual performance of the system depends on several parameters:

Juniper Networks recommends performing tests using a lab representative of the target production environment.

For standard deployments, 4 up to 8 physical cores are allocated. When HT siblings are not assigned for vRouter they MUST NOT be assigned for Nova or operating system because this can cause CPU steals.

In this diagram we can see that:

We can summarize this diagram performance result into following table:

In 2020, new tests have been proceeded (in Juniper and customer labs) with DPDK control threads vrouter service threads pinning. In these tests (done on RedHat OpeStack platform + contrail 19.12 + patch for DPDK control threads and vrouter service threads pinning) we’ve enforced CPU isolation (with isolcpus and tuned).

These tests have been done with 8 lcores and CPU siblings were used. vRouter physical interface was a bond of 2x10 Gb/s NIC (Intel Niantic cards).

MPPS per lcore is the number of packet per second that are processed at vrouter level. Given values are the performance seen at user level for a half duplex flow (outgoing only). For a full duplex flows (both outgoing and incoming direction) these values have to be divided by two.

On such a platform when packet size is more than 512 bytes we were reaching the physical interface maximum throughput. This is why performance per lcore seems to decrease at a very low rate for higher packet size.

We can use the results we’ve got for packet size below 512 bytes to extrapolate values for a higher packet size when we have more than 20 Gb/s available on vrouter physical interface.

In the diagram below, the MPPS per lcore values for 1024, 1280 and 1500 packet have been extrapolated using 64, 128, 256 and 512 packet size values :

This diagram could help our customers to evaluate the number of lcores needed to get a given performance at vrouter level.

The best results will be achieved when:

To enable Jumbo frames on tap and physical interfaces assign Jumbo on vhost0 interface.

On DPDK node the mtu needs to be set on vhost0 and it will update the pmd.

Before Contrail 3.2.8 and 4.0.1 on DPDK Compute:

There are limitations with jumbo frame support for Contrail Versions below 4.0.1 and 3.2.8:

DPDK Compute:

Kernel Compute:

This issue is tracked under https://bugs.launchpad.net/juniperopenstack/+bug/1731174 We do not support these offloads yet inside vrouter-dpdk.

Starting from Contrail 4.0.1 and 3.2.8, jumbo frames are supported in all scenarios.

Add parameters to configuration file /etc/contrail/contrail-vrouter-agent.conf

On every controller, add values AggregateInstanceExtraSpecFilter and NUMATopologyFilter to the scheduler_default_filters parameter in /etc/nova/nova.conf. Restart nova-scheduler.

Configure host aggregates:

that mem_page_size can have 1GB or 2048 values depending on hugepage size requirements of the VNF. Multiple flavors could be created in order to provide for both options.

Large pages are required for each instance (even any non-DPDK instance) running on hosts with DPDK vrouter. If large pages are not present in the guest, the interface will appear but will not function. This is why, on a compute node running DPDK vrouter, openstack instances must use flavors having hw:mem_page_size=large property.